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Background 


• Computational modeling and simulation (M&S) 
methods have substantial potential to support 
research, clinical decision and education in 
healthcare 

• Government agencies and industry are making 
substantial investments on R&D activities in 
simulation-based medicine and notable 
discoveries are being made 




Common practice guidelines do not exist to 

ensure these tools are appropriately applied 


Background 


To bridge this gap, the 

Committee on Credible Practice of 
Modeling & Simulation in Healthcare 

was established under 




& Multiscale Modeling (MSM) Consortium 




Definition of Simulation 


Credible Practice of Modeling & Simulation in Healthcare 


computational solution of models to quantify 
descriptive and predictive metrics of system(s) of 
interest; including related post-processing efforts 
to calculate these metrics from raw analysis results 


Charge: Overview 


GUIDELINES & For crecJ ifrl e P rac t“ ce in computational medicine by leveraging readily 

available techniques, and to define novel translational workflows to 
PROCEDURES enhance credibility practice* 


* 


DEMONSTRATE conducting studies for the implementation of novel credibility 
, w assessment procedures, and by disseminating examples of credibility 

Workflows assessmen t. 


* 


CONSISTENT To unify the use of M&S vocabulary for clear communication across a 
TERMINOLOGY variety of disciplines and stakeholders in the field. 


* 


Promote Good 
Practice 

By bridging synergistic activities of establishing confidence in 
simulation-based medicine conducted by M&S communities, as well 
as conducting outreach activities* 


End Product 

I. "Guidelines for Credible Practice of M&S in Healthcare" 

II. Proposed model certification process 

III* Identify new areas of research to advance 1 & II 






Charge: Implementation Strategy 


None of us are experts in everything . 
We need to learn from each other. 


Credible practice of modeling and simulation in healthcare 
requires ongoing inclusive communications to establish 
adaptive workflows that can be utilized broadly. 
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Charge: Implementation Strategy 


The Committee is divided into 
three Task Teams that represent 
the fundamental areas that are 
significant to our primary aims. 

This team-based structure helps 
to: 


• Establish a balanced 
representation of the interests 
and perspectives of the 
different stakeholders 

• Bridge synergistic activities in 
simulation-based medicine 
throughout the M&S 
communities 


Mathematical and 
Computational Sciences 


Team Coordinators 
(Committee Members) 
Jacob Barhak 
Joy Ku 

Marc Garbey 
Lu Tian 


AC Team Members 

Wing Kam Liu 
Anthony Hunt 
Vasilis Marmarelis 
Pras Pathmanathan 


End Users 


Team Coordinators 
(Committee Members ) 
Ahmet Erdemir 
Gary An 
Bill lytton 


AC Team Members 

James Thomas 
Ronald Germain 
David Eckmann 
Jeff Bischoff 


Practice Standards and 
Guidelines 


Team Coordinators 
(Committee Members) 
Lealem Mulugeta 
Tina Morrison 
Jerry Myers 


AC Team Members 

Martin Steel 
Marlei Walton 
Donna Lochner 
Grace Peng 


Cross-team Interaction (General Oversight by Co-Chairs) 


Objective: Represent the 
interests and perspectives 
of the M&S community 
consisting of fundamental 
and applied researchers, 
as well as commercial 
developers of healthcare 
M&S 


Objective: Represent the 
interests and perspectives 
of the M&S community 
that are mainly focused 
on using M&S for clinical 
research and practice. 


Objective: Represent the 
interests and perspectives 
of the M&S community 
that are focused on 
establishing professional 
and/or industry practice 
guidelines, regulations 
and standards. 


Global Stakeholders 


Government, Academia, Industry, and Multidisciplinary Professions 


■ 









Goal: Establish Credible Practice Guidelines 


The Committee’s primary deliverable by the end of the first two year 
term (03/2015) is to establish, “Guidelines for Credible Practice of 
Modeling and Simulation in Healthcare" 

Goal Oriented Activity: The CPMS Task Teams were charged to 
identify ten key elements or simple rules of credible practice in order 
to establish a foundation from which the “Guidelines for Credible 
Practice of Modeling and Simulation in Healthcare” can be 
developed. 

Full details of this activity is available at: 

http://wiki.simtk.ora/cpms/Ten Simple Rules of Credible Practice 


Ongoing: Ten Simple Rules of Credible Practice 


To initiate the Ten Simple Rules task, the following 26 simple rule candidates 
were initially generated and surveyed internally by the Committee. 


Use version control 
Use credible solvers 
Explicitly list your limitations 

Define the context the model is intended to 
be used for 

Define your evaluation metrics in advance 

Use appropriate data (input, validation, 
verification) 

Attempt validation within context 
Attempt verification within context 
Attempt uncertainty (error) estimation 

Perform appropriate level of sensitivity 
analysis within context of use 

Disseminate whenever possible (source 
code, test suite, data, etc) 

Report appropriately 

Use consistent terminology or define your 
terminology 


Get it reviewed by independent 
users/developers/members 

Learn from discipline-independent examples 
Be a discipline-independent/specific example 
Follow discipline-specific guidelines 
Conform to discipline-specific standards 
Document your code 
Develop with the end user in mind 

Provide user instructions whenever possible 
and applicable 

Practice what you preach 

Make sure your results are reproducible 

Provide examples of use 

Use traceable data that can be traced back 
to the origin 

Use competition of multiple implementations 
to check and balance each other 



Ongoing: Internal Survey Results Analysis 


# 

Consolidated & Rephrased Rules 

1 

Plan and develop the M&S with clear definition of the intended purpose or context, and with 
end-user input 

2 

Document the your code and the M&S (domain of validity/invalidity, intended use, user guide, 
etc), and make your code readable 

3 

Use version control 

4 

Test the M&S Appropriately within Context (V&V, UQ, sensitivity analysis (SA), test cases) 

5 

Use appropriate data 

6 

Disseminate whenever and whatever possible 

7 

Explicitly list the limitations of the M&S 

8 

Adopt and promote standard operating procedures 

9 

Use competition of multiple implementations to check and balance each other 

10 

Make it easy for anyone to reproduce and/or falsify your results 

11 

Use traceable data that can be traced back to the origin 

12 

Define your evaluation metrics in advance 

13 

Have the M&S reviewed by independent users/developers/members 


These results will be analyzed and reported in more detail in an upcoming 
presentation at the 2014 ASME V&V Symposium in May. 


Full details are available at: 

http://wiki.simtk.org/cpnns/Ten Simple Rules of Credible Practice/Summarv of Results 




Upcoming: Community and Model Surveys 


Simulation-Based Healthcare Community Survey: To ensure that we 
capture the perspectives and interests of the global stakeholder 
community, we are preparing to launch a public survey of the new 
33 simple rule candidates (increase due to internal survey results) 

Survey of Models: To complement the survey, we are also 
conducting a survey of computational models to develop a better 
understanding of the needs and successes of different types and 
applications of M&S in healthcare 

How can the SSH Community Participate? 

• We strongly encourage the SSH community to participate in 
and promote the survey 

• Contribute M&S examples that may help us better 
understand the needs and successes of different types and 
application of M&S within the SSH community 


Upcoming: Ten Simple Rules, Guidelines & Cert. 


Community Generated Ten Simple Rules 





Example: Fractional Flow Reserve - CT 


Description 

Credibility Assessment 

Intended use: Computes Fractional Flow 
Reserve (FFR) from non-invasive computed 
tomography images. 

Limitations: Clinical studies assess only the 
accuracy of the FFR CT quantity for coronary 
arteries 

Used clinical studies specifically designed 
to evaluate the efficacy of the model for 
non-invasive cardiac diagnostic relative to 
the traditional invasive FFR 

• HeartFlow NXT Study (201 3) - 254 patients 

• DeFACTO Study (2012) - 252 patients 

• DISCOVER-FLOW Study (201 1 ) - 1 03 
patients 

Appropriate Application 

Inappropriate Application 

• Determining FFR CT for clinical decision- 
making purposes in terms of coronary 
artery disease 

• Computing FFR CT for diagnosing of other 
parts of the vasculature 

• Suitable for teaching purposes centered 
around FFR (and FFR CT ) 

• Utilizing quantities from the simulation 
other than FFR CT (e.g., pressure, wall 
shear stress) for clinical decisions 


Example: Bone Fracture Risk Model 


Description 

Credibility Assessment 

Intended use: Probabilistic risk assessment of 
bone fracture due to bone changes due to 
zero-gravity exposure - analyses may be run 
for spaceflight or terrestrial scenarios 

Limitations: 

• Small n - “Attributable” data for model 
development 

• Uses DXA BMD, which has a limited 
correlation to bone strength or [ 
resistance to fracturel, and bone auality 

Compared to two published data sets that 
are considered to be among the latest state 
of knowledge in bone loss and risk of 
fracture 

• Development and Validation of a 
Predictive Bone Fracture Risk Model for 
Astronauts, Annals of Biomedical 
Engineering, 2009, Vol. 37, Number 1 1, 
2337-235 9 

Appropriate Application 

Inappropriate Application 

Provide supporting evidence for: 

• Bone fracture risk due to spaceflight 
related bone loss in terrestrial and 
spaceflight environment 

• Injury criteria definition: additional for 
fitness of duty evaluations for spaceflight, 
and injury loading thresholds 

• Changes to inflight injury likelihood 
resulting from exercise or pharmaceutical 
countermeasures to bone loss 

• As a sole source of evidence to make 
decisions regarding bone fracture risk — 
credibility level is not sufficient for this 

• Predict changes in bone volumetric 
density or architectural changes - |jQCT 
and finite element methods are required 
for this 


Example: Gross Cardiovascular Physiology 


Description 

Credibility Assessment 

Intended use: Education model to teach 
medical students about the general 
response of the cardiovascular system to 
various physiologic stresses, and 
administration of various interventions 

Limitations: Assumes average adult male 
between the ages of 25 and 55, weighing 
180 lbs, and 5’ 9” tall 

• Verification against simple test cases 

• Face/qualitative validation against 
literature data by physicians and 
physiologists for general trends in 
physiologic responses 

• Numerical stability and sensitivity analysis 
were not performed 

• Historical development data and versions 
were not documented 

Appropriate Application 

Inappropriate Application 

• A teaching tool to provide very high level 
insight of how the cardiovascular system 
may respond to various stressors and 
interventions built into the simulation 
environment 

• The educator should emphasize the 
simulations should be used as a 
supplement and not a replacement of 
their core curriculum 

• Conducting simulation scenarios that 
violate the primary assumptions 

• Simulations of other physiologic systems 

• Attempting to use the model for research 
or clinical purposes; credibility 
assessment is not sufficient for these kinds 
of applications, and it was developed for 
teaching purposes 

Note: This is a generalized example based off of a real use case of an educational model 



Goal: A Common Language Across Disciplines 


Goal Oriented Activity: A glossary of terms is being generated on 
the Committee’s to help unify the use of M&S vocabulary across a 
variety of disciplines and stakeholders in the field 


We strongly encourages all stakeholders (e.g. SSH community) to 
help establish these terms and definitions by visiting: 
http: / ! wiki. simtk.ora / coms/Glossarv and Definitions 


Model Clinical 



Example 


smikwiki-cmm Glossary and Definitions/ credibility 


FrontPage kccrmL^iinqr-: FmdPage Mc-ipCafttc-nts crctiihilrry 

Team 

.e Inin Attachments Mar* Actions 

frmwvtfkuKis- 


Pocumenb 

Credibility 

wiki 

Publications 

Dictionary Definition 

Public Fa rums 

Needs contnbiilion 

Adv-anced 

Committee Definition 

DnwinkMrik ft 

Source Cod* 

Needs contnbulion 

Thu pnaj«e hu no 
pybfcC dnwnl&irU. (»+ 
trikes ffrurce code 
jvaJabe 

Domain Specific Usage 


£ ngneenng and Biomedical (NASA) 1 he quality In eliol he Hat or Irust in M&S rasutlR. 


Notes 


1 from IMA5A STAN DARI ) l-QR MOPE L5 AND SIMULATIONS - '• htlpF. /fciandarcta nasa gow'0ocurnenlsrdeili8ib l 3i3-l53W 


Goal: Adoption of Guidelines by Stakeholders 


Goal Oriented Activity: We are taking every opportunity to engage 
and strongly encourage the global stakeholder community, such as 
the Society for SSH, to actively contribute to these efforts to ensure 
that the guidelines established capture the primary interests of the 
computational medicine community 

Open discussions and 
contribution to activities 
via: 

• Wiki pages 

• Discussion forum 

• Meeting minutes 

• Subversion repository 
access of all 
presentations, 
abstracts and posters 


ffcFWS 


URL: httgs^/simtlco^ 



Search Simtk.org 


I Go 


Advanced Search 
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Credible Practice of Modeling & Simulation in 
Healthcare 


Project Overview 
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Description: This 
project houses the 
activities of the 
Committee on 
Credible Practice of 
Modeling & Simulation 
in Healthcare. This is 
an initiative started at 
the Interagency 
Modeling and Analysis 
Group and Multiscale 
Modeling Consortium 

f http:// www.imagwiki.nibib.nih.gov j. Tentative charges of the 
Committee are to: 

i) adopt a consistent modeling & simulation terminology, 

ii) define accreditation procedures for modeling and simulation 
practice, 

iii) demonstrate accreditation workflows, and 


Project 

Lead 



Ahmet Erdemir 
Contact 



Lealem Mulugeta 
Contact 



Future Work 


• Launch public survey to the global community 
(~3 months) - Projected launch date: ??/2014 

• Analyze data, and publish results in PLOS 
Computational Biology in the form of “Ten Simple 
Rules of Credible Practice of M&S in Healthcare” 
- Estimated submission date: ??/2014 

• Develop and publish the “Guidelines for Credible 
Practice in Modeling & Simulation in Healthcare” 
based off of the Ten Simple Rules - Estimated 
completion date: 03/2015 


Open Floor Discussion 


• What are the areas that are most important to you in 
simulation-based healthcare? 


• Have we’ve missed something? 

• What are the challenges you face in applying 
simulations effectively to educate the healthcare 
practitioners? 

• Are there any use cases you want to discuss with us? 


Please take advantage of the survey, public forum and wiki site to 
continue this conversation with us. The guidelines will ultimately 
belong to you, so we want to hear from you! 




Committee on Credible Practice of 
Modeling & Simulation in Healthcare 

https://simtk. org/home/cpms 
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Definitions 


Credible Practice of Modeling & Simulation in Healthcare 


dependable with a desired certainty level to 

guide research or support decision making within 
a prescribed application domain and intended 
use; establishing reproducibility & accountability 


Definitions 


Credible Practice of Modeling & Simulation in Healthcare 


any activity involving development solution , 
interpretation and application of computational 
representation of biological, environmental and 
man-made systems and their interaction thereof 


Definitions 


Credible Practice of Modeling & Simulation in Healthcare 


specifically computational modeling; virtual 
representation of system(s) of interest in a usable 
form in order to provide descriptive and 
predictive metrics for timely and systematic 
exploration of the system (s) 


Definitions 


Credible Practice of Modeling & Simulation in Healthcare 


any activity involving development maintenance, 
advancement or administration of medical care; 

including research, diagnosis, risk assessment, 
prevention, therapy, rehabilitation, surgery, 
intervention design, and regulation 


Modeling for Coronary Artery 
Disease Diagnostics 



Fractional Flow 
Reserve (FFR) 

- Current gold 
standard in 
assessing coronary 
artery disease 

- Invasive procedure 

Heartflow's FFR CT 

technology 

computes FFR from 

non-invasive 

computed 

tomography 

images 


Video from http://heartflow.com 




Overview of FFR 
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computational fluid 
dynamics used to 
compute velocities and 
pressures, from which 


discretize model into 
mesh for computation 


FFR ct can then be 
derived 


Fonte, TA, Min, JK, "Computational Fluid Dynamics Applied to Cardiac Computed Tomography for Noninvasive Quantification of Fractional 
Reserve," 2013, JACC, 61(22):2233-2241 



Building Confidence in the Model 






Started out with simple, 
“easy-to-test" in vitro model 
over 10 years ago 



Methods developed and tested to 
address limitations in the interim 

Assessing what model can be 
used for 


A; Lumped-parameter heart model 



- Original idea: prediction of surgical 

bypass surgeries (to add: graph of results front 

- Recent clinical studies assess 
technology for coronary artery flow, 
not necessarily applicable to other 

fcfWSJ parts of the vasculature 



Bone Fracture Risk Model 


• What can we do to estimate astronaut risk of 
fracture? 

• Real and Present Concern: Skeletal Fracture 

- Weakened bones 

- Unique and off-nominal loading states 

• Lack of In Flight Injuries 

- Predictive data is limited 

• Fracture risk 

- Likelihood (unknown)+ Severity (known) 

• Our Question is: 

- What is the fracture likelihood in space (ISS, 
Orion) and on planetary activities (Moon and 
Mars)? 

- Can such assessments be extended to the BMD 
recovery period after return? 



The Loading Environment 
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Model Validation and Predictive Results 
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• Although femoral neck fracture 
is of high concern 

- Least likely location of fracture 

• Wrist most likely fracture 
location 


LS static lift 







0% A »AP«4G^ 

Femoral neck 
side fall 






& 


sr 




A 




A 


45° 

(EVA) 


45°(IVA) 90° (IVA) 


flexion angle 

Lumbar spine - Static 
lift 




or 



Up 


6.7m 


<> 



IVA 


<► 


<> 



EVA 


fall height Wrist impact 

Lumbar spine - Drop 
landing 


OMoon long (F) 
■ Mars short (M) 
♦ Mars short (F) 
□Mars long (M) 
OMars long (F) 



Highest 
sensitivities: 
EMU properties 


Nelson et al., 

Development and Validation of a 
Predictive Bone Fracture Risk 
Model for Astronauts, 

Annals of Biomedical Engineering, 
Vol 37, Number 11, 2337-2359 



